Recognition-based industrial automation control with confidence-based decision support

ABSTRACT

The present disclosure generally relates to a method for performing industrial automation control in an industrial automation system. As such, the method may include detecting, via a sensor system, positions and/or motions of a human. The method may then include determining a possible automation command corresponding to the detected positions and/or motions. After determining the possible automation command, the method may implement a control and/or notification action based upon the detected positions and/or motions.

BACKGROUND

The disclosure relates generally to systems and methods for controllingdevices within an industrial automation system. More particularly,embodiments of the present disclosure are related to systems and methodsfor controlling devices in the industrial automation system using rangecamera systems disposed within the industrial automation system.

Industrial automation systems are managed and operated using automationcontrol and monitoring systems. A wide range of applications exist forautomation control and monitoring systems, particularly in industrialautomation settings. Such applications may include the powering of awide range of actuators, such as valves, electric motors, and so forth,and the collection of data via sensors. Typical automation control andmonitoring systems may include one or more components, such as:programming terminals, automation controllers, input/output (I/O)modules, communication networks, and/or human-machine interface (HMI)terminals.

Generally, certain safety precautions are taken to ensure that devicesin industrial automation systems are operated safely. However,conventional industrial automation systems are limited in controllingits devices using the typical automation control and monitoring systemsdescribed above. Although these automation control and monitoringsystems may be used to manage the operations of the devices within theindustrial automation system, improved systems and methods for safelyoperating devices within an industrial automation system are desirable.

BRIEF DESCRIPTION

In one embodiment, a method for performing industrial automation controlin an industrial automation system may include detecting, via a sensorsystem, positions and/or motions of a human. The method may then includedetermining a possible automation command corresponding to the detectedpositions and/or motions. After determining the possible automationcommand, the method may implement a control and/or notification actionbased upon the detected positions and/or motions.

In another embodiment, a method for performing industrial automationcontrol may include detecting, via a sensor system, positions and/ormotions of an element of an industrial system. The method may thendetermine a possible automation action corresponding to the detectedpositions and/or motions. The method may then include implementing acontrol and/or notification action based upon the detected positionsand/or motions.

In yet another embodiment, a system for performing industrial automationcontrol may include a sensor system configured to detect positionsand/or motions of an element of an industrial system. The system mayalso include a processing component configured to compare the detectedpositions and/or motions with a plurality of automation commands, andidentify an automation command from the plurality of automation commandsbased at least in part on the comparison. The processing component maythen be configured to send the identified automation command to anindustrial automation device in the industrial system.

DRAWINGS

These and other features, aspects, and advantages of the presentinvention will become better understood when the following detaileddescription is read with reference to the accompanying drawings in whichlike characters represent like parts throughout the drawings, wherein:

FIG. 1 is a block diagram of a range camera system operating within anindustrial automation system, in accordance with an embodiment;

FIG. 2 is a block diagram of a range camera system employing atime-of-flight camera system operating within the industrial automationsystem of FIG. 1, in accordance with an embodiment;

FIG. 3 is a block diagram of a motion recognition system within therange camera system of FIG. 1, in accordance with an embodiment;

FIG. 4 is a flow chart of a method for controlling one or more deviceswithin the industrial automation system using the range camera system ofFIG. 1, in accordance with an embodiment;

FIGS. 5-6 are example visualizations of data received by the motionrecognition system of FIG. 3, in accordance with an embodiment;

FIG. 7 is a block diagram of a motion recognition engine and a positiondetermination engine within the motion recognition system of FIG. 3, inaccordance with an embodiment;

FIG. 8 is a block diagram of a machine operating within the industrialautomation system of FIG. 1, in accordance with an embodiment;

FIG. 9 is a flow chart of a method for controlling one or more deviceswithin the industrial automation system using the range camera system ofFIG. 1 and a secondary input, in accordance with an embodiment;

FIG. 10 is a flow chart of a method for controlling one or more deviceswithin the industrial automation system using derivative values ofpositions and/or motions detected by the range camera system of FIG. 1,in accordance with an embodiment;

FIG. 11 is a flow chart of a method for controlling one or more deviceswithin the industrial automation system based on whether humans orobjects are present in the industrial automation system as detected bythe range camera system of FIG. 1, in accordance with an embodiment;

FIG. 12 is a block diagram of a environment control system operating inconjunction with the motion recognition system of FIG. 3, in accordancewith an embodiment; and

FIG. 13 is a flow chart of a method for controlling one or more deviceswithin the industrial automation system based on distinguishing featuresbetween humans present in the industrial automation system as detectedby the range camera system of FIG. 1, in accordance with an embodiment.

DETAILED DESCRIPTION

The present disclosure is generally directed towards a range camerasystem disposed in an industrial automation system that may detectpositions and/or motions of persons and/or objects within a spaceencompassing the industrial automation system. After detecting thepositions and/or motions, the range camera system may perform variousoperations in the industrial automation system based on those detectedpositions and/or motions. That is, the range camera system may detectthe positions and/or motions of persons and/or objects (e.g., robots,autonomous guided vehicles, or machines) within the industrialautomation system space and use the detected positions and/or motions tocontrol various industrial automation devices in the industrialautomation system. In addition to detecting positions and/or motionswithin the industrial automation system, the range camera system maycompute a confidence value for each detected position and/or motion. Theconfidence value may relate to a degree or percentage in which theremote camera system may be certain in the detected position and/ormotion. As such, in certain embodiments, the range camera system maysend control signals to various devices in the industrial automationsystem based on the confidence values that correspond to the respectivedetected positions and/or motions. Additional details regarding therange camera system will be described below with reference to FIGS.1-13.

By way of introduction, FIG. 1 is a block diagram of an industrialautomation system 10 operating with a range camera system 12. The rangecamera system 12 may include one or more range cameras 14 and a motionrecognition system 16. The range camera 14 may include any type ofcamera or sensor system that may detect positions and/or movements ofany element in its viewing window. In certain embodiments, the rangecamera 14 may employ various types of technology to enable it to detectpositions and movement. By way of example, the range camera 14 mayinclude digital cameras, video cameras, infrared sensors, opticalsensors (e.g., video/camera), radio frequency energy detectors, soundsensors, sonar sensors, vibration sensors, magnetic sensors, and thelike to detect the positions and/or movements of any element in itsviewing window. The sensors used for the range camera 14 may be capableof detecting sonar waveforms, heat signatures, lasers, floor pressuresignatures, air pressure characteristics, and the like.

The range camera 14 may also include a sensor that may employ rangeimaging technology or techniques to produce two-dimensional orthree-dimensional images that may be used to indicate the distancebetween various points in an image acquired by the sensor and thelocation of the sensor. As such, the sensor employed with range imagingtechnology may operate according a number of techniques such as stereotriangulation, sheet of light triangulation, structured light, time-offlight, interferometry, coded aperture, and the like.

The detected positions and/or motions received by the range camera 14may be input into the motion recognition system 16, which may interpretthe detected positions and/or motions and determine various automationcommands that may correspond to the detected positions and/or motions.After determining the automation command that corresponds to thedetected positions and/or motions, the motion recognition system 16 maysend the automation command to the system controller 18, which mayimplement the automation command on an industrial automation device 20coupled to the system controller 18. For instance, the motionrecognition system 16 may receive a detected motion from the rangecamera 14 and may interpret the detected motion as a command to stop oneof the devices 20 in the industrial automation system 10. As such, themotion recognition system 16 may send a signal to the system controller18 to stop the respective industrial automation device 20.

Industrial automation devices 20 may include any type of machine ordevice that may operate within the industrial automation system 10.Examples of industrial automation devices 20 may include, but are notlimited to, actuators, electric motors, electric presses, and the like.Generally, the industrial automation devices 20 may be operated orcontrolled by the system controller 18. As such, the system controller18 may have a processor component, a memory component, a communicationcomponent, and the like such that it may operate the industrialautomation devices 20 by sending control signals to the industrialautomation devices 20. In certain embodiments, in addition to receivingautomation commands from the motion recognition system 16, the systemcontroller 18 may receive various types of information from varioussensors 22 in the industrial automation system 10. The sensors 22 may beany type of sensor such as a push-button sensor, a gas sensor, atemperature sensor, and the like that may be used to provide informationto the system controller 18. As such, the system controller 18 mayoperate the industrial automation devices 20 based on the informationreceived by the sensors 22, the automation commands interpreted by themotion recognition system 16, or both.

Keeping the foregoing in mind, in certain embodiments, the range camerasystem 12, the industrial automation devices 20, and the sensors 22 maygenerally be disposed in a space that may be characterized as part of anindustrial automation environment 24. The industrial automationenvironment 24 may include a physical space within the industrialautomation system 10 in which human operators 26, machines 28, ormaterial 30 may perform various actions to operate the industrialautomation system 10. As such, the range camera system 12 may monitorthe positions and/or motions of the humans 26, the machines 28, or thematerial 30 as they move within the industrial automation environment24.

The humans 26 may include any person that may be present in theindustrial automation environment 24. Generally, the humans 26 presentin the industrial automation environment 24 may include persons trainedand authorized to have access to the industrial automation environment24. The humans 26 may perform various operations such as maintenance andthe like for any device in the industrial automation system 10. However,it should be noted that the humans 26 may also include persons who arenot authorized or are undesirable persons (e.g., intruders) present inthe industrial automation environment 24. As such, in one embodiment,the range camera system 12 may be used to detect whether the humans 26in the industrial automation environment 24 are authorized to be in theindustrial automation environment 24 and send a notification orautomation command to the system controller 18 based on whether thedetected human(s) 26 is authorized or expected to be in the industrialautomation environment 24.

The machines 28 may include any automated or non-automated machinepresent in the industrial automation environment 24. As such, themachines 28 may include robots or mechanical devices that performvarious operations on items being manufactured, processed, or the like.For example, the machine 28 may include a mechanical claw that may movearticles within the industrial automation system. In certainembodiments, the machine 28 may include industrial automation devices 20operating within the view window of the range camera 14.

The material 30 may include items such as workpieces being moved by themachine 28 or the items being manufactured or processed in theindustrial automation system 10. For instance, FIG. 1 depicts thematerial 30 as a box on a conveyor belt. In this instance, the rangecamera system 12 may track the positions of the material 30 as it movesalong the conveyor belt. The motion recognition system 16 may then sendinformation related to the positions of the material 30 to the systemcontroller 18, which may use that information to control variousindustrial automation devices 20. For instance, if the system controller18 determines that the positions of the material 30 on the conveyor beltis changing too slowly, the system controller 18 may interpret thatinformation as indicating that the material 30 is moving too slowlyalong the conveyor belt. As such, the system controller 18 may send asignal to an industrial automation device 20, such as a motor drivecoupled to a motor that controls the movement of the conveyor belt, toincrease the speed of the motor, thereby increasing the speed at whichthe material 30 may move along the conveyor belt.

In certain embodiments, the range camera 14 may be coupled to variouscomponents in the industrial automation system 10 such as the industrialautomation device 20 or the like such that the range camera 14 maymonitor the movements and/or positions of various elements within theindustrial automation system 10. For instance, the range camera 14 maybe coupled to a DIN rail that may be fixed at various locations in theindustrial automation system 10, the industrial automation device 20, orthe like.

In any case, by using the range camera system 12 to monitor variouselements within the industrial automation system 10, the systemcontroller 18 may better manage the safe operations of the industrialautomation system 10. For instance, information acquired from the rangecamera system 12 may be used to determine the location of the human 26with respect to a potential hazard such as a moving industrialautomation device 20, a high-temperature industrial automation device20, high-voltage equipment, and the like. In certain embodiments, therange camera system 12 may process data related to body extremity of thehuman 26 that may be moving and the speed at which it may be moving topredict whether the movement may cause the human 26 to come in physicalcontact with a hazardous zone, a machine 28, or the like.

For instance, the range camera system 12 may process data received fromthe range camera 14 indicating the positions and movements of variousjoints of the human 26 to determine a probability of the human 26physically contacting or entering a hazardous zone. As such, the rangecamera system 12 may predict that the human 26 may come in contact withthe hazardous zone when the probability exceeds a threshold. By beingable to predict these types of situations, the range camera system 12may take preemptive measures in avoiding undesirable situations such astriggering an action to prevent the dangerous situation or minimize theadverse effects of the undesired situation. For example, the rangecamera system 12 may send the system controller 18 automation commandsto stop or alter the operations of the industrial automation devices 20or the machine 28 to such that the industrial automation devices 20 orthe machine 28 avoid the human 26. Moreover, the range camera system 12may also be used to restart or reengage the halted industrial automationdevices 20 or machine 28 once the human 26 is no longer located withinthe hazardous zone.

The range camera system 12 may also send a notification to the systemcontroller 18, other industrial automation systems 32, or the likeindicating that the human 26 was located in the hazardous zone or thatthe respective industrial automation devices 20 were stopped due thepresence of the human 26 in the hazardous zone. In this way, the otherindustrial automation systems 32 may modify their operations if it isdetermined that their operations will be affected by the data receivedby the motion recognition system 16.

In a similar fashion, the range camera system 12 may be used to monitorvarious the positions and/or motions of the machine 28 rather than, orin conjunction with, monitoring the positions and/or motions of thehuman 26. As such, the range camera system 12 may determine whether themachine 28 is operating safely within the industrial automationenvironment 24.

By way of example, FIG. 2 depicts an example of the industrialautomation environment 24 that uses a time-of-flight camera system 34 asthe range camera 14 to perform the various techniques described herein.As shown in FIG. 2, the time-of-flight camera system 34 may be employedin the industrial automation environment 24 or in a controlledenvironment such as a power control room. The time-of-flight camerasystem 34 may include an illumination unit 36 and an image sensor 37that may be used to determine the positions and/or motions of the human26, the machine 28, or the like present in the industrial automationenvironment 24 or in the controlled environment. In one embodiment, thetime-of-flight camera system 34 may rotate such that it may have a360-degree line-of-sight around the industrial automation environment24.

To determine the positions of the human 26 or the machine 28 in theindustrial automation environment 24, the time-of-flight camera system34 may use the illumination unit 36 to emit a light signal 38 outwardinto the industrial automation environment 24. The light signal 38 mayinclude an intensity-modulated light in the near-infrared range, alaser, or the like. In any case, once the light signal 38 hits anelement (i.e., the human 26 in FIG. 2), a reflected light signal 39 maybe reflected back to the time-of-flight camera system 34 and into theimage sensor 37. In one embodiment, the reflected light signal 39 may beprojected into the image sensor 37 via a lens that may be disposed onthe time-of-flight camera system 34. After sending the light signal 38and receiving the reflected light signal 39, the time-of-flight camerasystem 34 may send data related to the times at which the light signal38 was sent and the reflected light signal 39 was received to the motionrecognition system 16 to determine a distance between the element andthe time-of-flight camera system 34. That is, the motion recognitionsystem 16 may correlate data related to the light signal 38 and thereflected light signal 39 such as the times at which the light signal 38was sent and the reflected light signal 39 was received to determine adistance between the element and the time-of-flight camera system 34. Bytracking this distance over time, the motion recognition system 16 maydetermine the positions of the element over time, the motion of theelement, and the like.

In certain embodiments, the motion recognition system 16 may includethree-dimensional motion paths that may define the path in which themachine 28 may operate within the industrial automation environment 24.The three-dimensional motion paths may be determined based on anexpected motion of the machine 28 with respect to its function withinthe industrial automation system 10. As such, the motion recognitionsystem 16 may detect whether the machine 28 is operating as expectedwith respect to the three-dimensional motion paths and/or whether themachine 28 is experiencing any hazardous movements (e.g., moving tofast, proximate to human 26). If the motion recognition system 16detects that the machine 28 is operating unexpectedly or with certainhazardous movements, the motion recognition system 16 may send a signalto the system controller 18, which may be used to control the machine28, to operate the machine 28 in a safe state, powered off, or the like.

Moreover, in some embodiments, the motion recognition system 16 mayinterpret the motion and/or the speed of the motion of the machine 28 orthe human 26 to predict whether the machine 28 may enter a hazardousarea or hazardous situation such as contacting the human 26. As such,the motion recognition system 16 may send a signal to the systemcontroller 16 or directly to the machine 28 to alter the operation ofthe machine 28 such that the machine 28 may avoid entering the hazardousarea or encountering the hazardous situation. That is, the motionrecognition system 16 may control the operations of the machine 28 orany industrial automation device 20 based on the motions and/or thepositions of the machine, the motions and/or the positions of the human26, or the like, thereby operating the industrial automation system 10more safely.

Keeping the foregoing in mind, FIG. 3 illustrates a detailed blockdiagram 40 of the motion recognition system 16. The motion recognitionsystem 16 may include a communication component 42, a processor 44, amemory 46, a storage 48, input/output (I/O) ports 50, and the like. Thecommunication component 42 may be a wireless or wired communicationcomponent that may facilitate communication between the systemcontroller 18, the industrial automation devices 20, the range camera14, the machine 28, other industrial automation systems 32, and thelike. The processor 44 may be any type of computer processor ormicroprocessor capable of executing computer-executable code. The memory46 and the storage 48 may be any suitable articles of manufacture thatcan serve as media to store processor-executable code, data, or thelike. These articles of manufacture may represent computer-readablemedia (i.e., any suitable form of memory or storage) that may store theprocessor-executable code used by the processor 44 to perform thepresently disclosed techniques. Generally, the motion recognition system16 may receive motion and/or position data related to the human 26, themachine 28, and/or the material 30 and interpret the data to determineautomation commands for the industrial automation devices 20. The memory46 and the storage 48 may also be used to store the data, the respectiveinterpretation of the data, and the automation command that correspondsto the data. Although the block diagram 40 is depicted with respect tothe motion recognition system 16, it should be noted that the systemcontroller 18 may also include the same components to perform thevarious techniques described herein. Additional details describing amethod in which the motion recognition system 16 may use to interpretthe motion and/or location data will be discussed below with referenceto FIGS. 4-13.

Automation Commands Based on Detected Positions/Motions and ConfidenceValues

Referring now to FIG. 4, a method 60 for implementing automationcommands within the industrial automation system 10 based on detectedpositions and/or motions of an element within the viewable region of therange camera 14. In one embodiment, the techniques described herein withrespect to the method 60 may be performed by the motion recognitionsystem 16. As such, at block 62, the motion recognition system 16 mayreceive detected positions and/or motions of an element such as thehuman 26, the machine 28, or the material 30. That is, the range camera14 may receive data related to the positions and/or motions of theelement currently present within its viewing window.

In certain embodiments, the data related to the detected positionsand/or motions may include one or more optic images of the element, oneor more infrared images of the element, or the like. Once the rangecamera 14 acquires the images of the element, the motion recognitionsystem 16 may convert the images into two-dimensional figures thatrepresent the element. For instance, FIG. 5 illustrates an example of atwo-dimensional representation 80 of the human 26 and FIG. 6 illustratesan example of a two-dimensional representation 100 of the machine 28.

Referring briefly to FIG. 5, the two-dimensional representation 80 ofthe human 26 may include data points 82 at various edges or joints onthe body of the human 26. Based on the distance between the range camera14 and the human 26, the size of the two-dimensional representation 80,the relative distance between the two-dimensional representation 80 andother fixed objects within the view window of the range camera 14, andthe like, the motion recognition system 16 may generate metadata 84 foreach data point 82.

The metadata 84 may include certain details regarding the respectivedata point 82. By way of example, the metadata 84 may include anx-coordinate value 86, a y-coordinate value 88, a z-coordinate value 90,a confidence value 92, and the like. The confidence value 92 maycorrespond to a degree or amount in which the motion recognition system16 may be certain of the x-coordinate value 86, the y-coordinate value88, and the z-coordinate value 90. In some embodiments, the motionrecognition system 16 may generate the confidence value 92 for eachcoordinate value. The confidence value 92 may be determined based onvisibility of the data point 82, historical data related to the datapoint 82, or the like.

In the same manner, the two-dimensional representation 100 of themachine 28 may include data points 102 at various edges or joints on thebody of the machine 28. Based on the distance between the range camera14 and the machine 28, the size of the two-dimensional representation100, the relative distance between the two-dimensional representation100 and other fixed objects within the view window of the range camera14, and the like, the motion recognition system 16 may generate metadata104 for each data point 102.

Like the metadata 84 described above with reference to FIG. 5, themetadata 104 may include certain details regarding each respective datapoint 102 such as an x-coordinate value 106, a y-coordinate value 108, az-coordinate value 110, a confidence value 112, and the like. Like theconfidence value 92, the confidence value 102 may correspond to a degreeor amount in which the motion recognition system 16 may be certain ofthe x-coordinate value 106, the y-coordinate value 108, and thez-coordinate value 110. In some embodiments, the motion recognitionsystem 16 may generate the confidence value 112 for each coordinatevalue and may determine the confidence value 112 based on visibility ofthe data point 82, historical data related to the data point 82, anexpected positions of the machine 28, or the like.

Keeping the foregoing in mind, the motion recognition system 16 maydetect a motion or movement pattern of the element based on how thepositions of the data points 82 or 102 change over time. In oneembodiment, the motion recognition system 16 may acquire the metadata 84or 104 related to each data point 82 or 102 and store the metadata 84 or104 in the memory 46 or the storage 48. The processor 44 of the motionrecognition system 16 may then analyze the collection of metadata 84 or104 to determine how the element is moving. As such, the motionrecognition system 16 may recognize the motion or gestures of theelement.

Referring back to FIG. 4, at block 64, the motion recognition system 16may determine a possible automation command for the detected motionsreceived at block 62. In one embodiment, to determine the possibleautomation command, the motion recognition system 16 may compare thedetected positions and/or motions to a library of automation commandsrelated to detected positions and/or motions. FIG. 7 illustrates anexample of a library 122 of automation motion commands that may bestored in the storage 48 of the motion recognition system 16.

The library 122 may include a list of expected motions by the elementand a corresponding automation command. The list of expected motions maybe programmed into the library using a learning mode or the like withthe motion recognition system 16. As such, an operator or technician mayprovide an input to the motion recognition system 16 indicating anautomation command for a particular industrial automation device 20. Theoperator may then position himself within the view window of the rangecamera 14 and perform various movements or gestures that he intends tobe associated with the inputted automation command. As such, the motionrecognition system 16 may record how the data points 82 move or apattern of motion of the data points 82 that correspond to the human 26.The motion recognition system 16 may then store the pattern of motion ofthe data points 83 (automation commands 128) in the library 122 andassociated the pattern with the respective input automation command.

For example, the operator may provide an input to the motion recognitionsystem 16 to enter into a learning mode and specify that the aparticular motion or gesture should be associated with an emergency stopan automation command for a particular automation device 20. Afterreceiving these inputs, the motion recognition system 16 may detect themovements that correspond to the data points 82 of the operator, whichmay include, for instance, holding one arm out straight with a palm outand figures up, while the operator is in the view window of the rangecamera 14. Once the motion recognition system 16 detects the motion, themotion recognition system 16 may store the movement of the data points82 (automation command 128) with an association to an emergency stopautomation command in the library 122. In certain embodiments, themotion recognition system 16 may enter and exit the learning mode byreceiving some input from the operator that does not include anydetected motion or gesture. In this case, the configuration of thelearning mode may be secured and may not be compromised by anyinadvertent motions or gestures.

In certain embodiments, the processor 44 may include a motionrecognition engine 124 that may identify an automation command 128 fromthe library 122 based on the detected positions and/or motions of theelement. For example, if the motion recognition engine 124 detects thatthe human 26 is waving both of his arms left and right, the motionrecognition engine 124 may compare the detected motion to the motions orpatterns of motion stored in the library 122 and determine that themotion corresponds to a stop automation command for a particularindustrial automation device 20.

The library 122 may include a number of motions and a correspondingautomation command 128 for each motion. The automation commands mayinclude any command to control the automation devices 20, the machine28, or any other device in the industrial automation system 10. As such,the automation commands may include, but are not limited to, starting adevice, stopping a device, increasing a speed or output of a device,decreasing a speed or output of a device, and the like. Moreover, thelibrary 122 may include other commands associated with various motionssuch as disabling the motion recognition system 16, limiting the controlor ability of an operator to engage with the motion recognition system16, or the like. In certain embodiments, the library 122 may includemotions with respect to fixed objects in the industrial automationenvironment 24. For instance, a motion of the human 26 moving in thedirection of some control input device, such as an emergency stop input,may be associated with an automation command for that control inputdevice (e.g., emergency stop).

While the motion recognition engine 124 may be used to determine apossible automation command for the detected motions received at block62, a position determination engine 126 may be used to determine apossible automation command for the detected positions received at block62. As such, the position determination engine 126 may determine thepositions of the data points 82 and 102 and may consult the library 122to identify an automation command 128 that corresponds to the detectedpositions received at block 62. In this manner, the library 122 mayinclude rules or scenarios in which certain automation commands may beimplemented based on the positions of the data points 82 and 102.

Keeping this in mind, FIG. 8 illustrates a space 132 within theindustrial automation environment 24 that may be designated as hazardousareas or areas in which the human 26 should not enter or in which onlythe machine 28 should operate. As such, the library 122 may include anautomation command 128 to stop certain industrial automation devices 20that may potentially cause harm to the human 26 when the human 26 isdetected in the hazardous area.

For instance, in one embodiment, the space 132 may be marked by borders134, which may be implemented using a light curtain, electric markers,colored markers, or the like. As such, the library 122 may include anautomation command to stop industrial automation devices 20 or machines28 when one or more humans 26 are present within the space 132. Inanother example, the library 122 may include an automation command tostop the operation of the industrial automation device 20 or the machine28 when the human 26 or an extremity of the human 26 is located withinan operating space of the industrial automation device 20 or the machine28. For instance, if the industrial automation device 20 corresponds toan industrial press machine that compacts material, the library 122 mayinclude an industrial automation command that stops the operation of theindustrial press machine when the human 26 or an extremity of the human26 is detected within the area of the industrial press machine where thematerial is expected to be placed.

Referring back to FIG. 4, at block 66, the motion recognition system 16may determine the confidence value 92 or 112 for the detected positionsand/or motions received at block 62. As mentioned above, the motionrecognition system 16 may determine the confidence value 92 or 112 basedon visibility of the data point 82 or 102, historical data related tothe data point 82 or 102, expected positions of the human 26, themachine 28, or the material 30, or the like. In one embodiment, themotion recognition system 16 may also determine a confidence value forthe possible automation command determined at block 64 based on theconfidence value 92 or 112, the difference between the detectedpositions and/or motions determined at block 62 and the positions and/ormotions associated with the possible automation command received atblock 64, and the like. In another embodiment, the confidence value maybe related to the difference between the detected positions and/ormotions received at block 62 and an expected pattern of motion for theelement.

In certain embodiments, the motion recognition system 16 may determine aconfidence value for the detected positions and/or motions based onwhether the corresponding element is within a particular range. Forinstance, if the element that corresponds to the detected positionsand/or motions is within a close range of the camera 14, the detectedpositions and/or motions may be associated with higher confidence valuesas compared to elements that may be further from the camera 14. Theconfidence values may also be stored in an array that may be organizedbased on a distance between the detected element and the camera 14. Assuch, the motion recognition system 16 may determine a confidence valuefor a detected positions and/or motions based on a confidence value aslisted in the array.

At block 68, the motion recognition system 16 may determine whether theconfidence value determined at block 66 is greater than some threshold.If the confidence value determined at block 66 is greater than thethreshold, the motion recognition system 16 may proceed to block 70 andimplement the automation command determined at block 64. In addition toor in lieu of implementing the automation command, the motionrecognition system 16 may send a notification to an administrator oroperator of the industrial automation system 10 indicating that theautomation command is being implemented or the like.

When implementing the determined automation command, in one embodiment,the motion recognition system 16 may associate the determined automationcommand with a particular industrial automation device 20. That is, upondetermining the automation command based on the detected positionsand/or motions of the element, the motion recognition system 16 maydetermine which automation device 20 or machine 28 to implement thedetermined automation command based on the positions of the data points82 and 102 within a proximity of a device in the industrial automationsystem 10. For example, if the motion recognition system 16 interpretsmotions detected by the range camera 14 to correspond to stopping anindustrial automation device 20, the motion recognition system 16 maythen, at block 70, implement the automation command on the industrialautomation device 20 that is within the closest proximity to theelement.

Referring back to block 68, if the confidence value determined at block66 is not greater than the threshold, the motion recognition system 16may proceed to block 72 and implement a default automation command. Thedefault automation command may be a safe state command for respectiveautomation device 20 or machine 28. In one embodiment, the defaultautomation command may include continuing the operation of therespective industrial automation device 20. In another embodiment, atblock 72, the motion recognition system 16 may send a notificationrequesting a manual confirmation of the automation command determined atblock 64 or a notification indicating that the confidence level is lowfor the respective automation command.

Automation Commands Based on Detected Positions/Motions and SecondaryInput

By determining whether to implement the recognized automation commandbased on the confidence value, the motion recognition system 16 may besafeguarded or protected against interpreting random movements and/orpositions of the human 26, the machine 28, or the material 30 aspossible automation commands. However, in certain embodiments, to betterensure that the appropriate automation command is being recognized bythe motion recognition system 16 or to better ensure that the industrialautomation devices 20 are being operated safely, the motion recognitionsystem 16 may use secondary inputs to supplement, complement, or evenreplace motion-based and/or position-based inputs recognized by themotion recognition system 16. For example, FIG. 9 depicts a method 140for controlling one or more industrial automation devices 20 within theindustrial automation system 10 using the range camera system 12 andsome secondary input.

The secondary input may include any input or signal received orgenerated by the system controller 18 or the motion recognition system16 that may be used to complement or confirm the automation command. Assuch, the motion recognition system 16 may verify the validity of theinterpreted automation command based on whether an expected secondaryinput has been received by the system controller 18 or the motionrecognition system 16.

Keeping this in mind and referring to FIG. 9, at block 142, the motionrecognition system 16 may receive detected positions and/or motions ofthe element, as described above with respect to block 62 of FIG. 4. Inone embodiment, the detected positions and/or motions of the elementreceived by the motion recognition system 16 may include metadata orinformation related to the confidence value 92 or 112 for the detectedpositions and/or motions received at block 142.

After receiving the detected positions and/or motions of the element, atblock 144, the motion recognition system 16 may determine a possibleautomation command for the detected positions and/or motions received atblock 142, as described above with respect to block 64 of FIG. 4.Although not shown in FIG. 9, it should be noted that in certainembodiments, the motion recognition system 16 may determine a confidencevalue for the automation command determined at block 144 based on theconfidence value 92 or 112, the difference between the detectedpositions and/or motions received at block 142 and expected positionsand/or motions that correspond to the possible automation commanddetermined at block 144, and the like.

At block 146, the motion recognition system 16 may determine whether anexpected secondary input or automation system signal has been received.As mentioned above, the secondary input may include an input or signalreceived by the motion recognition system 16 to verify that theautomation command determined at block 144 corresponds to an automationcommand to be implemented. As such, the secondary input may include theuse of manual or other control inputs such as a push button locatedwithin the industrial automation system 10, an input provided to themotion recognition system 16 via the input/output ports 50, an inputreceived by the motion recognition system 16 via a sensor disposed onthe industrial automation device 20 or within the industrial automationsystem 10, or the like.

For example, the secondary input may include a manual switch or buttonoperated by the human 26. The secondary inputs may also include a weightdetected on a safety/pressure mat, a signal from a light curtainindicating whether the light curtain has been broken, an indication froman area scanner, or inputs from enable pendants, safety gates, guardinterlocks, emergency stop switches, or the like.

Another example of the secondary input may include voice commands orconfirmations. As such, at block 144, the machine recognition system 16may expect to receive a voice command from the human 26 that maycorrespond to an expected voice command for the respective determinedautomation command. For instance, the expected secondary inputassociated with a detected motion that may correspond to a stopautomation command may include a speech by the human 26 that recites,“stop.”

In one embodiment, the secondary input may be the detected positions ofthe element with respect to some automation device 20, while theautomation command determined at block 144 may be based on the detectedmotions of the element. For example, if the motion recognition system 16determines that the motions of the human 26 correspond to an automationcommand that engages a press machine, the motion recognition system 16may, at block 146, determine whether the detected location of the human26 is a safe distance away from the press machine or outside of an areain which a material may be pressed. As such, the location of the human26 may be used as the expected secondary input at block 146. Moreover,in this manner, the motion recognition system 16 may coordinate motionsdetected by the range camera 14 with positions detected by the rangecamera 14 to control the operation of various devices within theindustrial automation system 10. As such, the motion recognition system16 may ensure that industrial automation devices 20 are operated safelyand as per their respective specifications.

Referring back to block 146, if the motion recognition system 16receives the expected secondary input, the motion recognition system 16may proceed to block 148 and implement the automation command determinedat block 144. In one embodiment, in addition to or in lieu ofimplementing the automation command, the motion recognition system 16may send a notification message (e.g., e-mail, text message) to anadministrator or operator of the industrial automation system 10 thatindicates that the automation command has been implemented, that theautomation command has been requested to be implemented, or the like.

If, however, at block 146, the motion recognition system 16 does notreceive the expected secondary input, the motion recognition system 16may proceed to block 150 and implement a default action. In oneembodiment, a default action may be defined or determined for eachautomation device 20. The default action may defined or determined tocorrespond to a safe-sate (e.g., de-energized) for the respectiveindustrial automation device 20. In other words, if the motionrecognition system 16 does not receive the expected secondary input, themotion recognition system 16 may determine the safe-state for therespective industrial automation device 20 and implement the automationcommand that corresponds to the safe-state. For example, if the expectedsecondary input corresponds to a signal indicating that the human 26 isstanding on a pressure mat and therefore located a safe distance awayfrom the respective industrial automation device 20, the motionrecognition system 16 may determine the safe-state automation commandfor the respective automation device 20 by assuming that the human 26 isnot standing on the pressure mat. As such, the motion recognition system16 may determine that the safe-state for the respective automationdevice 20 may include de-energizing the respective automation device 20,thereby ensuring that a person is not hurt or an object is not damagedby the respective automation device 20. In this case, the motionrecognition system 16 may then send a command to the respectiveautomation device 20 to operate in the safe-state mode. In certainembodiments, the default action or the safe-state mode may includeperforming no action at all. As such, the default action may includecontinuing the operation of the respective automation device 20.

In addition to or in lieu of implementing the default action, the motionrecognition system 16 may send a notification to an administrator oroperator of the industrial automation system 10 that indicates that thedefault action has been implemented, that the secondary input has notbeen received, or the like. In one embodiment, the notification mayinclude suggestions or instructions as to what the motion recognitionsystem 16 may be seeking as the secondary input.

In certain embodiments, after determining that the expected secondaryinput has been received, the motion recognition system 16 may proceed toblock 66 of FIG. 4. As such, the motion recognition system 16 maydetermine the confidence value 92 or 112 for the detected positionsand/or motions received at block 142. Alternatively or additionally, themotion recognition system 16 may determine a confidence value for thepossible automation command determined at block 144, as described abovewith respect to block 66. The motion recognition system 16 may thenfollow the process of the method 60 to may better ensure that theautomation command determined at block 144 may correctly match thedetected positions and/or motions of the element received at block 142.

Automation Commands Based on Detected Positions/Motions Derivative Data

In addition to determining automation commands based on detectedpositions and/or motions of an element, the motion recognition system 16may also analyze a derivative value of the detected positions and/ormotions when determining automation commands. That is, the motionrecognition system 16 may determine one or more derivative values of thedetected positions for an element and determine an automation commandbased on the derivative values. The derivative values of datacorresponding to the detected positions of the element may indicate avelocity in which the element is moving. In certain embodiments, themotion recognition system 16 may use the velocity information indetermining an automation command for the detected positions. Forinstance, the velocity data may be used to quantify a speed in which thedetermined automation command should be implemented.

In the same manner, the derivative values of data corresponding to thedetected motions of the element may indicate an acceleration or a changein acceleration that corresponds to how the element may be moving. Theacceleration or change in acceleration data may be interpreted by themotion recognition system 16 to indicate a sense of urgency or anundesired event with respect to the detected motions of the element. Forinstance, if the human 26 moves with increasing speed towards anemergency stop switch, the motion recognition system 16 may interpretthe high acceleration or high change in acceleration data thatcorresponds to the motion of the human 26 as indicating that a seriousissue may be present in the industrial automation system 10. Moreover,the acceleration or change in acceleration data may also be used topredict a location of the human 26 (or other tracked object) at variouspoints in time. Additional details with regard to using derivative-baseddata in determining automation commands for the industrial automationsystem 10 is provided below with reference to FIG. 10.

Referring now to FIG. 10, the motion recognition system 16 may employ amethod 160 to implement automation commands and/or notifications basedon derivative data related to positions and/or motions detected by therange camera 14. At block 162, the motion recognition system 16 mayreceive detected positions and/or motions of an element, as describedabove with reference to block 62 of FIG. 4.

After receiving the detected positions and/or motions of the element, atblock 164, the motion recognition system 16 may determine derivativevalues for the detected positions and/or motions received at block 162.In one embodiment, for detected position data, the motion recognitionsystem 16 may generate a position function that represents the positionsof the data points 82 or 102 over time. The motion recognition system 16may then determine a derivative value for the generated positionfunction to determine a rate of change in the detected positions of thedata points 82 or 102, which may correspond to a velocity in which thedata points 82 or 102 move.

In another embodiment, for detected motion data, the motion recognitionsystem 16 may generate a motion function that represents the motions ofthe data points 82 or 102 over time. The motion recognition system 16may then determine a derivative value for the generated motion functionto determine a rate of change in the detected motions of the data points82 or 102, which may correspond to an acceleration of the detectedmotion for the data points 82 or 102.

At block 166, the motion recognition system 16 may determine possibleautomation commands based on the detected positions and/or motionsreceived at block 162 and/or the derivative values of the detectedpositions and/or motions determined at block 164. That is, the motionrecognition engine 124 may determine an automation command thatcorresponds to the detected positions and/or motions from the library122, as described above with reference to block 64 of FIG. 4, and mayrefine or revise the determined automation command based on thederivative values of the detected positions and/or motions determined atblock 164.

For example, in addition to the data related to positions and/or motionsof an element, the library 122 may also include expected velocities andaccelerations that correspond to the data related to the positionsand/or motions of the element. In other words, in addition toassociating certain positions and/or motions of an element with aparticular automation command, the library 122 may also associate thecertain positions and/or motions of the element of the particularautomation command with expected velocities and/or acceleration valuesthat correspond to how the element moves. In this manner, the motionrecognition system 16 may detect a sense of urgency in the automationcommand or may flag any unusual behavior of the element that invoked theautomation command.

Keeping this in mind, at block 166, the motion recognition system 16 mayfirst retrieve an automation command from the library 122 based on thedetected positions and/or motions of the element. However, the motionrecognition 16 may then revise or alter the retrieved automation commandbased on the derivative values of the detected positions and/or motions.By way of example, after determining that the positions and/or motionsof an element correspond to an automation command that increases thespeed of a motor, the motion recognition system 16 may then consult thelibrary 122 to determine whether the derivative values of the detectedpositions and/or motions of the element are within expected ranges ofthe derivative values of the detected positions and/or motions for theelement.

If the derivative values of the detected positions and/or motions of theelement are not within the expected ranges of the derivative values, themotion recognition system 16 may alter or modify the automation commandretrieved from the library 122. For instance, referring back to theexample above, if the automation command corresponds to increasing thespeed of a motor, the motion recognition system 16 may change theautomation command to disable the motor. In this example, the motionrecognition system 16 may disable the industrial automation device 20(i.e., motor) to ensure that the industrial automation device 20 isoperated safely and to ensure that the appropriate element's positionsand/or motions were detected. That is, by recognizing whether thederivative values relatively matched expected the expected derivativevalues, the motion recognition system 16 may provide additional securityin preventing unauthorized personnel from operating devices in theindustrial automation system 10.

In certain embodiments, the library 122 may generally include datarelated to expected velocity and acceleration values for motions thatcorrespond to each of the automation commands stored therein. Inaddition to expected velocity and acceleration values for automationcommands, the library 122 may also store expected velocity andacceleration values that correspond to the general behavior of anelement such as the human 26, the machine 28, or the material 30. Assuch, the expected velocity and acceleration values may include a rangeof velocity and acceleration values that correspond to normal behavioralmovements of the human 26, the machine 28, or the material 30. Incertain embodiments, the motion recognition system 16 may determine theexpected velocity and acceleration values for any element based onhistorical data acquired by the range camera 14. That is, the motionrecognition system 16 may detect the positions and/or motions of variouselements over time and identify positions and/or motions of the elementsthat may correspond to normal behavior. The motion recognition system 16may then determine the derivative values for the positions and/ormotions of the elements that correspond to the normal behavior andidentify a range of derivative values (e.g., velocity and accelerationdata) correspond to the normal behavior.

Keeping this in mind, the motion recognition system 16 may, at block166, determine an automation command based on whether the detectedpositions and/or motions have derivative values that correspond to theexpected derivative values. That is, the motion recognition system 16may interpret whether the detected motions of the element correspond tonormal movements or abnormal movements and determine an automationcommand based on whether the element's movements are normal or abnormal.For instance, if the human 26 suddenly falls to the floor or becomesimmobile, the motion of the human 26 falling to the floor or remainingimmobile may have derivative values that are outside a range of expectedderivative values for a human. As such, the motion recognition system 16may recognize or detect these types of undesirable situations based onthe velocity or acceleration data related to the motion of the human 26.In another example, the motion recognition system 16 may determine thatthe machine 28 or the material 30 may be moving at a velocity oracceleration that may fall outside of the expected range of derivativevalues. In this case, the motion recognition system 16 may determinethat the machine 28 or the material 30 may be moving abnormally, whichmay result in damage to the machine 28, the material 30, or the like.

If the motion recognition system 16 determines the derivative valuesthat are outside a range of expected derivative values, the motionrecognition system 16 may then determine one or more automation commandsfor various industrial automation devices 20 that may be designed tokeep the human 26 safe, the industrial automation system 10 functioning,or the like. For instance, the motion recognition system 16 may predictthe movements and operations of the industrial automation devices 20that may be within a proximity of the human 26 or that may risk furtherinjuring the human 26 and determine automation commands that mayminimize any risk of further injury to the human 26. For example, thedetermined automation command(s) may include stopping the operation ofvarious industrial automation devices 20, machines 28, or the like.

In certain embodiments, the motion recognition system 16 may, at block166, determine possible automation commands based on the derivativevalues by predicting whether the human 26, the machine 28, the material30, or the automation device 20 may physically contact each other orenter within each other's operating space. That is, using the derivativevalues of the detected motions received at block 162, the motionrecognition system 16 may predict whether two elements are moving in apattern that may result in an injury or damage to either element.Moreover, by using the detected positions and/or motions of the elementand the derivative values of the detected positions and/or motions, themotion recognition system 16 may predict when the elements may collide,cause injury, cause damage, or the like. As such, the motion recognitionsystem 16 may determine one or more automation commands that may preventinjury or damage to any element within the industrial automation system10. For instance, if the motion recognition system 16 predicts that twoelements may collide with each other based on the detected positionsand/or motions of the elements and the derivative values of the detectedpositions and/or motions, the motion recognition system 16 may determineautomation commands for various industrial automation devices 20 and/ormachines 28 that may cause the industrial automation devices 20 and/ormachines 28 to move in an opposite direction than its current directionto avoid the collision.

In some embodiments, the motion recognition system 16 may quantify thedetermined automation command based on the derivative values of thedetected positions and/or motions. In other words, the motionrecognition 124 may attribute some numerical value to an aspect thatrelates to the determined automation command. For example, if the motionrecognition engine 124 retrieves an automation command from the library122 that corresponds to stopping the operation of an industrialautomation device 20, the motion recognition system 16 may use thederivative value of the detected positions and/or motions to quantify asense of urgency or importance related to the automation command to stopthe industrial automation device 20. That is, if the derivative valuesindicate that the velocity and/or acceleration are greater than anexpected range of velocity and/or acceleration, the motion recognitionsystem 16 may determine that the corresponding automation command has ahigh degree of urgency or importance. As such, at block 166, the motionrecognition system 16 may determine additional automation commands basedon the heightened sense of importance. Referring back to the exampleabove, after retrieving an automation command from the library 122 thatcorresponds to stopping the operation of an industrial automation device20, the motion recognition system 16 may generate additional automationcommands to stop other related industrial automation devices 20 toprotect the entire industrial automation system 10 if the retrievedautomation command is interpreted to have a high importance.

After determining the possible automation command based on the detectedpositions and/or motions and/or based on the derivative values, themotion recognition system 16 may implement the automation command(s)determined at block 166. In some cases, the motion recognition system 16may send a notification to an administrator or operator of theindustrial automation system 10 that indicates the determined automationcommand(s). If the motion recognition system 16 detected an abnormal orundesirable event, as described above, the motion recognition system 16may send a notification to the administrator or operator of theindustrial automation system 10 that provides details related to theabnormal or undesirable event.

Referring back to block 166, in certain embodiments, after determiningthe possible automation command(s), the motion recognition system 16 mayproceed to block 66 of FIG. 4. As such, the motion recognition system 16may determine the confidence value 92 or 112 for the detected positionsand/or motions received at block 162. Alternatively or additionally, themotion recognition system 16 may determine a confidence value for thepossible automation command determined at block 166, as described abovewith respect to block 66. In this manner, the motion recognition system16 may better ensure that the automation command determined at block 166may correctly match the detected positions and/or motions of the elementreceived at block 162.

In another embodiment, after determining the possible automationcommand(s) at block 166, the motion recognition system 16 may proceed toblock 146 of FIG. 9. As such, the motion recognition system 16 maydetermine whether an expected secondary input has been received, asdescribed above with reference to block 146 of FIG. 9. The motionrecognition system 16 may then verify the validity of the determinedautomation command based on whether an expected secondary input has beenreceived by the system controller 18 or the motion recognition system16.

Referring again to block 166, the motion recognition system 16 maydetermine possible automation commands based on historical data that maybe stored in a memory. That is, the motion recognition system 16 maylearn to associate particular patterns of motion with a respectiveautomation command using the historical data. For instance, if themotion recognition system 16 determines that a particular pattern ofmotion is typically associated with a particular automation commandaccording to the historical data, the motion recognition system 16 maylearn or associate the particular pattern of motion with the particularautomation command.

Automation Commands Based on Detected Positions/Motions and Type ofElement Present

In addition to interpreting positions and/or motions of an element inthe industrial automation system as described above in FIG. 4, FIG. 9,and FIG. 10, the motion recognition system 16 may also distinguishbetween humans and objects in the industrial automation environment 24and implement various controls and/or notification actions based onwhether the element is human or not (FIG. 11). For example, the motionrecognition system 16 may control how lights and air conditioning withinthe industrial automation environment 24 may function based on whether ahuman or an object is moving in the industrial automation environment24.

Keeping this in mind, FIG. 11 depicts a method 170 for implementingvarious control and/or notification actions related to the industrialautomation system 10 based on whether a human or an object (i.e.,non-human) is present in the industrial automation environment.Referring now to FIG. 11, at block 172, the motion recognition system 16may receive detected positions and/or motions of an element present inthe industrial automation environment 24 from the range camera 14, asdescribed above.

At block 174, the motion recognition system 16 may determine whether theelement corresponds to a human (e.g., the human 26) or an object (e.g.,the machine 26 or the material 30) based on the detected positionsand/or motions received at block 172. In certain embodiments, the motionrecognition system 16 may compare the positions of the data points 82 or102 with templates or known data point arrangements of humans. If thepositions of the data points 82 or 102 correspond to the template orknown data point arrangements of humans, the motion recognition system16 may determine that the element is a human. Otherwise, the motionrecognition system 16 may classify the element as an object ornon-human.

In another embodiment, the motion recognition system 16 may compare themotions of the element received at block 172 with templates or knownmotion patterns of humans. If the received motions correspond to thetemplate or known motion patterns of humans, the motion recognitionsystem 16 may determine that the element is a human. Otherwise, themotion recognition system 16 may classify the element as an object ornon-human.

Once the motion recognition system 16 determines that the element iseither a human or an object, the motion recognition system 16 mayimplement a control action and/or a notification action based on whetherthe element is determined to be a human or an object. The control actionmay include operating any device in the industrial automation system 10or controlling various environmental parameters in the industrialautomation environment 24. For instance, FIG. 12 illustrates an exampleenvironment control system 180 that may use the motion recognitionsystem 16 to control the various environmental parameters in theindustrial automation environment 24, as described above with referenceto the method 170.

Referring now to FIG. 12, the environment control system 180 may includevarious environment control features such as a heating, ventilation, andair conditioning (HVAC) unit 182, a light control unit 184, or the like.The HVAC unit 182 may control the airflow, maintain the air quality, orregulate the temperature in the industrial automation environment 24 byproviding ventilation, air filtration, and the like. The light controlunit 184 may control the operations of lights 186 used to illuminate theindustrial automation environment. In certain embodiments, the HVAC unit182 and the light control unit 184 may be coupled to the systemcontroller 18 as shown in FIG. 12. As such, the system controller 18 mayprovide signals to the HVAC unit 182 and the light control unit 184 tocontrol the environment within the industrial automation environment 24.However, it should be noted that in some embodiments, the motionrecognition system 16 may be directly coupled to the environment controldevices, as opposed to via the system controller 18.

Keeping the environment control system 180 in mind and referring back toblock 176, the motion recognition system 16 may control the operationsof the HVAC unit 182, the light control unit 184, or the like based onwhether the detected element is a human or an object. For instance, theconditions within the industrial automation environment 24 should notcater to humans when humans are not present therein. As such, in oneembodiment, if the motion recognition system 16 determines that theelement present in the industrial automation environment 24 is not ahuman, the motion recognition system 16 may send one or more signals tothe system controller 18 to modify the operations of the HVAC unit 182,the light control unit 184, and the like. By way of example, the motionrecognition system 16 may send signals to the system controller 18 tostop operating the HVAC unit 182 or turn the lights 186 off when themotion recognition system 16 determines that no human is present in theindustrial automation environment 24. In this manner, the industrialautomation devices 20, the machines 28, and the material 30 may continueto move throughout the industrial automation environment 24 whileoperating in a human-unsuitable (e.g., uncomfortable) condition orlow-light condition. That is, since the industrial automation devices20, the machines 28, and the material 30 may not be affected byoperating in a human-suitable or human-unsuitable environment, themotion recognition system 16 may operate the environment control devicesmore efficiently by minimizing the use of the HVAC unit 182 or the lightcontrol unit 184. Moreover, once the received detected positions and/ormotions of the element are determined to be human, the motionrecognition system 16 may, at block 176, adjust the operations of theenvironment control devices to make the industrial automationenvironment 24 more suitable or comfortable for humans.

Besides controlling the environment control devices in the industrialautomation system, the motion recognition system 16 may also control theindustrial automation devices 20 or the machines 28 based on whether theelement is human or not. For instance, in certain embodiments, themotion recognition system 16 may be arranged such that the range camera14 may monitor an area within the industrial automation environment 24,which may be designated as a hazardous area or an area where humansshould not enter. As such, if, at block 174, the motion recognitionsystem 16 determines that the element in the area corresponds to ahuman, the motion recognition system 16 may send one or more signals tothe system controller 18 to alter or stop the operation of theindustrial automation devices 20, the machines 28, or the like toprotect the human now present in the area. In this manner, the motionrecognition system 16 may allow a wide variety of objects of varioussizes and orientations to pass into a hazardous area, while maintaininga safe environment in the industrial automation environment 24 when ahuman is present.

The notification actions may include sending a notification message(e.g., e-mail, text message) to an administrator or operator of theindustrial automation system 10 that indicates that the automationcommand has been implemented, that the control action of block 176 hasbeen implemented. In one embodiment, the notification message may alsoinclude an indication that a human is present in the industrialautomation environment 24 when it is not authorized to be present andthe like.

In addition to implementing control actions and/or notification actionsbased on whether the detected positions and/or motions are made byhumans or non-humans, the motion recognition system 16 may alsoimplement control actions and/or notification actions based distinctionsdetected between humans present in the industrial automation environment24. For instance, FIG. 13 depicts an example of a method 190 forimplementing automation commands and/or notification actions based ondistinctions detected between humans in the industrial automationenvironment.

Referring now to FIG. 13, at block 192, the motion recognition system 16may scan the industrial automation environment 24 using the range camera14. At block 194, the motion recognition system 16 may identify humansoperating or present in the industrial automation environment 24. Themotion recognition system 16 may then determine which of the presenthumans are authorized to implement automation commands using the motionrecognition system 16.

In one embodiment, the human 26 may perform a particular motion orgesture to designate himself as an authorized person in the industrialautomation environment 24. As such, the human 26 may use a gesture-basedsign in motion or authentication process to designate himself as anauthorized person.

In another embodiment, the motion recognition system 16 may search forindividuals wearing a particular article of clothing such as a vest, abutton, a belt, gloves, or the like. The particular article of clothingmay designate the individual as the authorized human for implementingautomation commands. In some instances, humans carrying or wearing anundesirable object such as a gun or laptop may be designated asunauthorized personnel. In these instances, the motion recognitionsystem 16 may send a notification or alarm to indicate the presence ofunauthorized personnel in the industrial automation environment 24.

In yet another embodiment, the motion recognition system 16 may searchfor an emblem or symbol disposed on the human 26 such as a badge todesignate the human 26 as the authorized human. In some instance, themotion recognition system 16 may work in conjunction with the sensors 22to detect a radio frequency identification (RFID) tag or the likedisposed on the human 26 to designate the human 26 as the authorizedhuman.

The motion recognition system 16 may also store profiles of authorizedindividuals in its memory. For instance, the profiles may include datarelated to facial features, body parameters, or the like associated withauthorized humans. As such, the motion recognition system 16 may, atblock 194, identify authorized personnel based on whether their facialfeatures, body parameters, or the like match that of the facialfeatures, body parameter, or the like stored in the profiles.

The motion recognition system 16 may also recognize an authorized humanbe receiving a gesture or motion from a known authorized humanindicating that another human is also authorized. That is, a knownauthorized human may transfer his authorization status to another humanor designate another human in the industrial automation environment 24as authorized using particular motions, gestures, or the like.

The motion recognition system 16 may also identify authorized humans bydetecting specific color or clothing attributes on humans. In oneembodiment, different colors or clothing attributes disposed on a humanmay indicate different levels of authority. That is, different levels ofauthority may be associated with different humans in the industrialautomation environment 24. Each different level of authority may enablethe respective human to implement a different set of automationcommands. Although the different levels of authority has been describedas being associated on humans based on detecting specific colors orclothing attributes, it should be noted that the motion recognitionsystem 16 may designate different humans as having different levels ofauthority based on particular motions or gestures performed by a human,particular articles of clothing worn by the human, certain emblems,symbols, or tags being detected on the human, stored profiles of thehuman, as described above.

After the motion recognition system 16 identifies the authorizedpersonnel at block 194, the motion recognition system 16 may, at block196, receive detected positions and/or motions that correspond to theidentified authorized personnel from the range camera 14. In certainembodiments, the motion recognition system 16 may proceed to block 66 ofFIG. 4. As such, the motion recognition system 16 may then proceed toblock 64 and follow the process of the method 60 to implement automationcommands using confidence values.

After receiving the detected positions and/or motions of the authorizedhuman at block 196, the motion recognition system 16 may proceed toblock 144 of FIG. 9 and follow the process of the method 140. As such,the motion recognition system 16 may determine whether an expectedsecondary input has been received, as described above with reference toblock 146 of FIG. 9. As such, the motion recognition system 16 mayverify the validity of the determined automation command of theauthorized human based on whether an expected secondary input has beenreceived by the system controller 18 or the motion recognition system16.

In one embodiment, after receiving the detected positions and/or motionsof the authorized human at block 196, the motion recognition system 16may proceed to block 164 of FIG. 10 and follow the process of the method160. As such, the motion recognition system 16 may determine theautomation command that corresponds to the detected positions and/ormotions based on derivative values for the detected positions and/ormotions, as described above with reference to block 166 of FIG. 10. Assuch, the motion recognition system 16 may further interpret thedetected positions and/or motions of the human 26 using the derivativevalues.

While only certain features of the invention have been illustrated anddescribed herein, many modifications and changes will occur to thoseskilled in the art. It is, therefore, to be understood that the appendedclaims are intended to cover all such modifications and changes as fallwithin the true spirit of the invention.

The invention claimed is:
 1. A method for performing industrialautomation control in an industrial automation system, comprising:detecting, via a sensor system, one or more gestures of a human, a firstset of positions of the human, and a second set of positions of anindustrial automation device; determining a probability that the humanwill contact the industrial automation device based on the first andsecond sets of positions; determining a possible automation commandcorresponding to the detected one or more gestures; implementing a firstcontrol action based upon the possible automation command, wherein thecontrol action is configured to cause an automation controller tocontrol an operation of the industrial automation device; implementing asecond control action based upon when the probability exceeds a value,wherein the second control action comprises adjusting the operation ofthe industrial automation device to avoid contacting the human.
 2. Themethod of claim 1, wherein the one or more gestures correspond to aplurality of possible gestures, each gesture being associated with adetermined automation command.
 3. The method of claim 1, comprising:determining a confidence value associated with the detected one or moregestures and the possible automation command; and implementing the firstcontrol action based upon the confidence value.
 4. The method of claim3, comprising implementing a third control action configured to adjustthe operation of the industrial automation device in the industrialautomation system when the confidence value is below a threshold.
 5. Themethod of claim 3, wherein the confidence value is indicative ofrelative accuracy in the determination of at least one of the detectedgestures.
 6. The method of claim 3, wherein the confidence value isindicative of relative accuracy of matching of the one or more gesturesto the first automation command.
 7. The method of claim 3, wherein theconfidence value is indicative of relative accuracy of matching of theone or more gestures to an expected pattern of movement.
 8. The methodof claim 3, wherein implementing the first control action comprisessending the determined automation command to the industrial automationdevice when the confidence value is greater than a threshold.
 9. Themethod of claim 1, comprising sending a notification message indicativeof the determined automation command.
 10. A method for performingindustrial automation control, comprising: detecting, via a sensorsystem, gestures of an element of an industrial system; detecting, viathe sensor system, positions and/or motions of a human operating in theindustrial system; detecting, via the sensor system, positions and/ormotions of a machine operating in the industrial system; determining apossible automation action corresponding to the detected gestures;determining a probability that the human will contact the machine basedon the positions and/or motions of the human and the positions and/orthe motions of the machine; implementing a control action based upon thepossible automation action, wherein the control action is configured tocause an automation controller to control an operation of an industrialautomation device; and sending a command to alter the operation to causethe machine to avoid contacting the human when the probability isgreater than a threshold.
 11. The method of claim 10, wherein theelement comprises a machine component.
 12. The method of claim 10,wherein the element comprises a workpiece.
 13. The method of claim 10,wherein the control action comprises a command for operation of themachine.
 14. The method of claim 10, comprising: determining aconfidence value associated with the detected gestures and thedetermined automation action; and implementing the control action basedupon the confidence value.
 15. The method of claim 14, comprisingsending a notification message indicating that the confidence value islow when the confidence value is not greater than a limit.
 16. Themethod of claim 10, wherein the command to alter the operation comprisesstopping the operation of the machine.
 17. A system for performingindustrial automation control, comprising: a sensor system configuredto: detect gestures of an element of an industrial system; detectpositions and/or motions of a human operating in the industrial system;and detect positions and/or motions of a machine operating in theindustrial system; a processing component configured to: compare thedetected gestures with a plurality of automation commands; identify anautomation command from the plurality of automation commands based atleast in part on the comparison; determine a probability that the humanwill contact the machine based on the positions and/or motions of thehuman and the positions and/or the motions of the machine; send theidentified automation command to the machine in the industrial systemafter the automation command is identified; send a command to alter theidentified automation command to cause the machine to avoid contactingthe human when the probability is greater than a threshold.
 18. Thesystem of claim 17, wherein the sensor system comprises digital cameras,video cameras, infrared sensors, optical sensors, radio frequency energydetectors, sound sensors, vibration sensors, magnetic sensors, heatsensors, pressure sensors, sonar sensors, range imaging technology, orany combination thereof.
 19. The system of claim 17, wherein theprocessing component is configured to: determine whether the element iswithin a space of the industrial system; and send a second command to atleast one industrial automation device in the industrial system to stopat least one operation of the at least one industrial automation devicewhen the element is within the space.
 20. The system of claim 19,wherein the space is designated using a light curtain.
 21. The system ofclaim 17, wherein the processing component is configured to generate aconfidence value that corresponds to the detected gestures and based atleast in part the comparisons, wherein the identified automation commandis sent to the industrial automation device based at least in part onthe confidence value.
 22. The system of claim 21, wherein the confidencevalue is generated based at least in part on visibility of the element,historical data related to the element, expected positions of theelement, or any combination thereof.
 23. The system of claim 21, whereinthe processing component is configured to send a second automationcommand to the industrial automation device to operate in a safe statewhen the confidence value is not greater than a threshold.
 24. Thesystem of claim 23, wherein the safe state comprises continuing theoperation of the industrial automation device.